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DETAILED ACTION 
Response to Amendment 

Applicant's submission (request for reconsideration) filed 04/25/2005 has been entered. 
Claims 1, 3-4, 6-10, 12-19, 24-29 and 31-32 are pending in the application. 



Response to Arguments 

Applicant's arguments with respect to claims 1, 3-4, 6-10, 12-19, 24-29 and 31-32 have been 
considered but are found not persuasive in view of the ground(s) of rejection set forth in the last 
Office Action based on Daily U.S. Patent No. 6,317,127 (Daily). 

As set forth in the subsequent rejection, Daily teaches the claim limitations of the 
amended claim 1 and similar claims. Applicant argues on Page 3 of Request for Reconsideration 
that Daily fails to disclose or suggest more than one video sensing means for sensing a video of a 
real space from different viewpoints. However, it is known that Daily discloses a multi-sensor 
system 152 to capture a wide FOV video signals in combination with each user's head tracker 44 
(a sensing means) for sensing a video of a real space from different viewpoints because multiple 
users can sense a video of a real space from different viewpoints using the combination of the 
sensing means of the multi-sensor system 152 and the individual user's sensing means. 
Therefore, Daily teaches or suggests the claim limitation of more than one video sensing means 
for sensing a video of a real space from different viewpoints (user's viewpoints). Daily's system 
supports multiple views simultaneously as Daily points out that the prior art has a drawback of a 
system limited to a single user and does not support multiple views simultaneously. Moreover, 
Daily discloses a local processor 48 responds to the current user FOV 46 by extracting that 
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portion of the video signal 22 that lies in the user FOV and its registered symbols 26, maps it into 
the user's coordinate system and transmits the subimage 42 to the HMD 40 and the local 
processor 48 includes a subimage capture unit 50 that downloads the data from the video and 
overlay distribution busses 24 and 36 and stores it in a buffer. Daily further discloses that the 
subimage capture unit preferably downloads an oversized subimage such that the tables used to 
. extract the data off the buses do not have to be updated for small changes in the user's FOV and 
a viewpoint display unit 52 reads out the video and overlay data from the buffer for the current 
user FOV and transmits the subimage to the HMD. 

Daily teaches an augmented reality presentation apparatus for superimposing a virtual 
object in a real space, characterized by comprising: objective viewpoint augmented reality 
presentation means for superimposing the virtual object viewed from a first viewpoint position, 
which differs from any player's viewpoint position, in the real space viewed from the first 
viewpoint position (e.g., figure 1, 9-10; column 3-5; column 10-12), wherein said objective 
viewpoint augmented reality presentation means includes first video sensing means for sensing a 
video of the real space viewed from the first viewpoint position wherein CCD cameras or a 
multi-sensor system 152 attached to the underside of the airplane to image the hemisphere below 
the plane, the GPS 154 providing the plane *s current position in coordinates and a second 
sensor array placed on top of the airplane for viewing star constellations (figure 1, 9-10; column 
3-5; column 10-12); first video generation means for generating a video of the virtual object 
viewed from the first viewpoint position wherein a source processor 16 includes individual 
channel processors 18 that group the pixels into packets and a multiplexer 20 that multiplexes 
the packets into a wide FOV video signal 22 and the overlay distribution processor 24 interacts 
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with the positioning system 32, the database 30 to determine which overlay symbols and audio 
tracks lie within the current wide FOVand the subimage capture unit 50 downloads the data 
from the viewo and overlay distribution buses 24 and 26 and stores it in a buffer (figure 7, 9-10; 
column 3-5; column 10-12); first video composition means for composing an augmented reality 
video viewed from the first viewpoint position on the basis of the videos of the real space and the 
virtual object viewed from the first viewpoint position wherein a source processor 16 includes 
individual channel processors 18 that group the pixels into packets and a multiplexer 20 that 
multiplexes the packets into a wide FOV video signal 22 and the overlay distribution processor 
24 interacts with the positioning system 32, the database 30 to determine which overlay symbols 
and audio tracks lie within the current wide FOVand the subimage capture unit 50 downloads 
the data from the viewo and overlay distribution buses 24 and 26 and stores it in a buffer (figure 
1, 9-10; column 3-5; column 10-12); and objective viewpoint video display means for displaying 
the augmented reality video obtained from said first video composition means, the objective 
viewpoint video display unit being separate from any display unit adapted to display to any 
player an augmented reality video viewed from a player's viewpoint position wherein the 
multiplexer 20 that multiplexes the packets into a wide FOV video signal 22 and the overlay 
distribution processor 24 interacts with the positioning system 32, the database 30 to determine 
which overlay symbols and audio tracks lie within the current wide FOVand the subimage 
capture unit 50 downloads the data from the viewo and overlay distribution buses 24 and 26 and 
stores it in a buffer; A viewpoint display unit 52 reads out the video and overlay data from the 
buffer for the current user FOVand transmits the subimage to the HMD wherein the display 52 
is separate and di fferent from the display within HMD; see figure 1 } 9-10; column 3-5; column 
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70-72. Daily further teaches player's viewpoint augmented reality presentation means for 
superimposing the virtual object viewed from the player's viewpoint position in the real space 
viewed from the player's viewpoint position (e.g., figure 7, 9-70; column 3-5; column 70-72); 
wherein said player's viewpoint augmented reality presentation means includes second video 
sensing means for sensing a video of the real space viewed from the player's viewpoint position 
including an external stereo camera setup slaved to the user 's head movement so that the 
cameras track his head movement and provide the desired view of the surroundings; column 1; 
and the tracking device 164 such as an RF beacon or a gyro compass to follow the user 's head 
movement and output coordinates that define the users FOV, or the viewpoint position including 
a head tracker 44 for tracking the user *s head movement in spherical coordinate to select a user 
FOV 45 from the wide FOV video signal (See figure I, 9-10; column 3-5; column 10-12)\ 
player's viewpoint position acquiring means for acquiring information indicating the player's 
viewpoint position wherein an external stereo camera setup slaved to the user 's head movement 
so that the cameras track his head movement and provide the desired view of the surroundings; 
column 1; and the tracking device 164 such as an RF beacon or a gyro compass to follow the 
user 's head movement and output coordinates that define the users FOV, or the viewpoint 
position including a head tracker 44 for tracking the user 's head movement in spherical 
coordinate to select a user FOV 45 from the wide FOV video signal (see figure 1, 9-10; column 
3-5; column 10-12)', second video generation means for generating a video of the virtual object 
viewed from the player's viewpoint position using the information indicating the player's view 
position wherein the distribution processor 24 uses the position and heading information 
including the appropriate geographical range, class and hierarchical levels of symbols and 
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audio tracks for each user to obtain a subimage 42 of the wide FOV video signal 22 and a local 
processor 48 responds to the current user FOV 46 by extracting that portion of the video signal 
22 that lies in the user FOV and its registered symbols 26, maps it into the user 's coordinate 
system and transmits the subimage 42 to the HMD 40. The local processor 48 includes a 
subimage capture unit 50 that downloads the data from the video and overlay distribution busses 
24 and 36 and stores it in a buffer. The subimage capture unit pre ferably downloads an 
oversized subimage such that the tables used to extract the data off the buses do not have to be 
updated for small changes in the user 's FOV and a viewpoint display unit 52 reads out the video 
and overlay data from the buffer for the current user FOV and transmits the subimage to the 
HMD (see (figure I, 9- JO; column 3-5; column 10-12)\ second video composition means for 
composing an augmented reality video viewed from the player's viewpoint position on the basis 
of the videos of the real space and the virtual object viewed from the player's viewpoint position 
wherein the distribution processor 24 uses the position and heading information including the 
appropriate geographical range, class and hierarchical levels of symbols and audio tracks for - 
each user to obtain a subimage 42 of the wide FOV video signal 22 and a local processor 48 
responds to the current user FOV 46 by extracting that portion of the video signal 22 that lies in 
the user FOV and its registered symbols 26, maps it into the user *s coordinate system and 
transmits the subimage 42 to the HMD 40. The local processor 48 includes a subimage capture 
unit 50 that downloads the data from the video and overlay distribution busses 24 and 36 and 
stores it in a bu ffer. The subimage capture unit pre ferably downloads an oversized subimage 
such that the tables used to extract the data o ff the buses do not have to be updated for small 
changes in the user 's FOV and a viewpoint display unit 52 reads out the video and overlay data 
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from the buffer for the current user FOVand transmits the subimage to the HMD (see figure 1, 
9-10; column 3-5; column 10-12); and display means for displaying to the player the augmented 
reality video viewed from the player's viewpoint position (e.g., HMD; figure 7, 9-10; column 3- 
5; column 10-12). 

Claim Rejections - 35 USC §102 
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

1. Claims 1, 3-4, 6-10, 12-18, 28-29, and 31-32 are rejected under 35 U.S.C 102(e) as being 
anticipated by Daily U.S. Patent No. 6,317,127 (Daily). 

2. Claim 1: 

Daily teaches an augmented reality presentation apparatus for superimposing a virtual 
object in a real space, characterized by comprising: 

Objective viewpoint augmented reality presentation means for superimposing the virtual 
object viewed from a first viewpoint position, which differs from any player's viewpoint 
position, in the real space viewed from the first viewpoint position (e.g., figure 1,9-10; column 
3-5; column 10-12), wherein said objective viewpoint augmented reality presentation means 
includes 
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First video sensing means for sensing a video of the real space viewed from the first 
viewpoint position (e.g., CCD cameras or a multi-sensor system 152 attached to the underside of 
the airplane to image the hemisphere below the plane, the GPS 154 providing the plane 's 
current position in coordinates and a second sensor array placed on top of the airplane for 
viewing star constellations; figure 1, 9- JO; column 3-5; column 10-12)', 

First video generation means for generating a video of the virtual object viewed from the 
first viewpoint position (e.g., A source processor 16 includes individual channel processors 18 
that group the pixels into packets and a multiplexer 20 that multiplexes the packets into a wide 
FOV video signal 22 and the overlay distribution processor 24 interacts with the positioning 
system 32, the database 30 to determine which overlay symbols and audio tracks lie within the 
current wide FOV and the subimage capture unit 50 downloads the data from the viewo and 
overlay distribution buses 24 and 26 and stores it in a buffer; figure 1, 9-10; column 3-5; column 
10-12)- 

First video composition means for composing an augmented reality video viewed from 
the first viewpoint position on the basis of the videos of the real space and the virtual object 
viewed from the first viewpoint position (e.g., A source processor 16 includes individual channel 
processors 18 that group the pixels into packets and a multiplexer 20 that multiplexes the 
packets into a wide FOV video signal 22 and the overlay distribution processor 24 interacts with 
the positioning system 32, the database 30 to determine which overlay symbols and audio tracks 
lie within the current wide FOV and the subimage capture unit 50 downloads the data from the 
viewo and overlay distribution buses 24 and 26 and stores it in a buffer; figure 1, 9-10; column 
3-5; column 10-12\ and 
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Objective viewpoint video display means for displaying the augmented reality video 
obtained from said first video composition means, the objective viewpoint video display unit 
being separate from any display unit adapted to display to any player an augmented reality video 
viewed from a player's viewpoint position (e.g., the multiplexer 20 that multiplexes the packets 
into a wide FOV video signal 22 and the overlay distribution processor 24 interacts with the 
positioning system 32, the database 30 to determine which overlay symbols and audio tracks lie 
within the current wide FOV and the subimage capture unit 50 downloads the data from the 
viewo and overlay distribution buses 24 and 26 and stores it in a buffer; A viewpoint display unit 
52 reads out the video and overlay data from the buffer for the current user FOV and transmits 
the subimage to the HMD wherein the display 52 is separate and different from the display 
within HMD : see figure 1, 9- JO; column 3-5; column 10-12); 

wherein said apparatus further comprises: 

Player's viewpoint augmented reality presentation means for superimposing the virtual 
object viewed from the player's viewpoint position in the real space viewed from the player's 
viewpoint position {e.g., figure 1, 9-10; column 3-5; column 10-12); 

wherein said player's viewpoint augmented reality presentation means includes 
Second video sensing means for sensing a video of the real space viewed from the 
player's viewpoint position (e.g f an external stereo camera setup slaved to the user *s head 
movement so that the cameras track his head movement and provide the desired view of the 
surroundings; column 1; and the tracking device 164 such as an RF beacon or a gyro compass 
to follow the user 's head movement and output coordinates that define the users FOV, or the 
viewpoint position; figure 1, 9-10; column 3-5; column 10-12); 
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Player's viewpoint position acquiring means for acquiring information indicating the 
player's viewpoint position (e.g., an external stereo camera setup slaved to the user 's head 
movement so that the cameras track his head movement and provide the desired view of the 
surroundings; column 1; and the tracking device 164 such as an RF beacon or a gyro compass 
to follow the user 's head movement and output coordinates that define the users FOV, or the 
viewpoint position; figure 1, 9-10; column 3-5; column 10-12); 

Second video generation means for generating a video of the virtual object viewed from 
the player's viewpoint position using the information indicating the player's view position (e.g., 
the distribution processor 24 uses the position and heading information including the 
appropriate geographical range, class and hierarchical levels of symbols and audio tracks for 
each user to obtain a subimage 42 of the wide FOV video signal 22; figure 1, 9-10; column 3-5; 
column 10-12)', 

Second video composition means for composing an augmented reality video viewed from 
the player's viewpoint position on the basis of the videos of the real space and the virtual object 
viewed from the player's viewpoint position (e.g., the distribution processor 24 uses the position 
and heading information including the appropriate geographical range, class and hierarchical 
levels of symbols and audio tracks for each user to obtain a subimage 42 of the wide FO V video 
signal 22; figure 1, 9-10; column 3-5; column 10-12); and 

Display means for displaying to the player the augmented reality video viewed from the 
player's viewpoint position (e.g., HMD; figure 1, 9-10; column 3-5; column 10-12). 



Claim 3: 
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The apparatus according to claim 1, characterized in that said player *s viewpoint 
augmented reality presentation means further comprises: the second video generation means for 
generating a video of the virtual object viewed from said player *s viewpoint position; and the 
display means for displaying to the player the video of the virtual object viewed from said 
player 's viewpoint position on a display surface through which the player can visually see the 
real space. 

Claim 3 recites all the limitations of claim 1 and adds the limitation of "the second video 
generation means" and "the display means." Daily teaches a video generation means and display 
means (e.g., figure 1, 9-10; column 3-5; column 10-12), 

Claim 4: 

The apparatus according to claim 1, characterized by further comprising information 
generation means for generating information that pertains to rendering of the virtual object, and 
in that said first video generation means and said second video generation means generate 
videos of the virtual object using the information that pertains to rendering of the virtual object 

Claim 4 recites all the limitations of claim 1 and adds the limitation of "information 
generation means" and "generation means generate videos." Daily teaches an information 
generation means (e.g., figure 1, 9-10; column 3-5; column 10-12). 

Claim 6: 

The apparatus according to claim 1, characterized in that parameters of said first video 
sensing means are known, and said first video generation means generates the video of the 
virtual object viewed from said first viewpoint position in accordance with the known 
parameters. 
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Claim 6 recites all the limitations of claim 1 and adds the limitation of "the known 
parameters." Daily teaches the known parameters such as the focal length of the CCD video 
camera and measured plane's coordinates by GPS sensors, and input parameters such as the 
desired resolution, the object's focal length, orientation and Cartesian coordinates on the object's 
image plane where the object is either the image source or the user display {figure 1, 9-10; 
column 3-5; column 7-12). 

Claim 7: 

The apparatus according to claim 1, characterized in that some of parameters of said 
first video sensing means are variable, said apparatus further comprises measurement means for 
measuring changes of the parameters, and said first video generation means generates the video 
of the virtual object viewed from said first viewpoint position in accordance with the parameters 
measured by said measurement means. 

Claim 7 recites all the limitations of claim 1 and adds the limitation of "variable 
parameters" and "measurement means." Daily teaches measuring the plane's positions are 
variables of the plane's coordinates in which the coordinate values are measured by the GPS 
sensors {figure 1, 9-10; column 3-5; column 7-12). 

Claim 8: 

The apparatus according to claim 7, characterized in that the parameters of said first 
video sensing means measured by said measurement means include at least one of a viewpoint 
position/posture, and zoom ratio. 
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Claim 8 recites all the limitations of claim 7 and adds the limitation of "position/posture 
information." Daily teaches the plane's positions are variables of the plane's coordinates which 
determine the position/posture of the CCD video cameras and the zoom ratios are determined by 
the zooming in and out parameters inputted by the user. Daily also teaches generating and 
correcting view position and posture of the passengers {figure 1, 9-10; column 3-5; column 7- 
12), 

Claim 9: 

The apparatus according to claim 1, characterized in that when a plurality of first video 
sensing means equivalent to said first video sensing means are present, said apparatus further 
comprises selection means for receiving a plurality of videos of the real space from said first 
viewpoint position from the plurality of first video sensing means, and outputting a video of the 
real space viewed from said first viewpoint position from one selected first video sensing means 
to said first video composition means, and said first video composition means generates a video 
of the virtual object viewed from said first viewpoint position using parameters of the first video 
sensing means selected by said selection means. 

Claim 9 recites all the limitations of claim 1 and adds the limitation of "selection means 
for receiving a plurality of first video sensing means." 

Daily teaches selecting of the image regions as captured by a plurality of video cameras 
by zooming in and out the video scene in the real space. Daily also teaches a plurality of video 
cameras for sensing a scene below or around the plane where image signals that represent an 
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environment scene of a real space and the user's zooming in and out video scene (figure l t 9-10; 
column 3-5; column 7-12). 



3. Claim 10: 

The claim 10 is a rephrasing of the claim 1 in a method form. The claim is rejected for 
the same reason as set forth in claim 1. 

Claim 12: 

The apparatus according to claim 10, characterized in that said player 's viewpoint 
augmented reality presentation step further comprises: the second video generation step of 
generating a video of the virtual object viewed from said player 's viewpoint position; and the 
display step of displaying to the player the video of the virtual object viewed from said player 's 
viewpoint position on a display surface through which the player can visually see the real space. 

Claim 12 recites all the limitations of claim 10 and adds the limitation of "the second 
video generation step" and "the display step." 

Daily teaches the second video generation step and the display step including the video 
generation in accordance to the user's zooming in and out parameters and the HMD for viewing 
the video scene (figure 1, 9-10; column 3-5; column 7-12), 



Claim 13: 
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The method according to claim 10, characterized by further comprising information 
generation step of generating information that pertains to rendering of the virtual object, and in 
that said first video generation step and said second video generation step generate videos of the 
virtual object using the information that pertains to rendering of the virtual object. 

Claim 13 recites all the limitations of claim 10 and adds the limitation of "information 
generation step" and "first and second generation steps to generate videos." 

Daily teaches the information generation step and the first and second generation steps to 
generate videos {figure l t 9-10; column 3-5; column 7-12). 

Claim 14: 

The apparatus according to claim 13, characterized in that said information generation 
step includes the step of generating information of an outer appearance of the virtual object and 
information of a position/posture of the virtual object as the information that pertains to 
rendering of the virtual object 

Claim 14 recites all the limitations of claim 13 and adds the limitation of "generating 
information of an outer appearance of the virtual object and information of a position/posture of 
the virtual object." 

Daily teaches capturing the plane's coordinates, generating and correcting view position 
and posture of the passengers, zooming in and out the video signal, and selecting the local FOV 
thereby correcting the viewpoint position/posture of the virtual object (figure l t 9-10; column 3- 
5; column 7-12). 

Claim 15: 
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The method according to claim 10, characterized in that parameters of said first video 
sensing means are known, and said first video generation step includes the step of generating the 
video of the virtual object viewed from said first viewpoint position in accordance with the 
known parameters. 

Claim 15 recites all the limitations of claim 10 and adds the limitation of "the known 
parameters." Daily teaches the known parameters such as the focal length of the CCD video 
camera and measured plane's coordinates by GPS sensors, and the video scenes can be adjusted 
in accordance to the desired resolution, the object's focal length, orientation and Cartesian 
coordinates on the object's image plane where the object is either the image source or the user 
display (figure 1, 9-10; column 3-5; column 7-12). 

Claim 16: 

The method according to claim 10, characterized in that some of parameters of means for 
sensing a video viewed from said first viewpoint position are variable, said method further 
comprises the measurement step of measuring changes of the parameters, and said first video 
generation step includes the step of generating the video of the virtual object viewed from said 
first viewpoint position in accordance with the parameters measured in the measurement step. 

Claim 16 recites all the limitations of claim 10 and adds the limitation of "variable 
parameters" and "measurement step." Daily teaches measuring the plane's coordinates which are 
variables as plane moves. Daily further teaches correcting view position and posture of the 
passengers and zooming in and out the video scene {figure 1, 9-10; column 3-5; column 7-12). 
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Claim 17: 

The method according to claim J 6, characterized in that the parameters of the means for 
sensing a video viewed from said first viewpoint position measured in the measurement step 
include at least one of a viewpoint position/posture, and zoon ratio. 

Claim' 17 recites all the limitations of claim 16 and adds the limitation of 
"position/posture information " Daily teaches the plane's positions are variables of the plane's 
coordinates which determine the position/posture of the CCD video cameras and the zoom ratios 
are determined by the zooming in and out parameters inputted by the user. Daily also teaches 
generating and correcting view position and posture of the passengers {figure 1, 9-10; column 3- 
5; column 7-12). 

Claim 18: 

The method according to claim 10, characterized in that when a plurality of means for 
sensing a video viewed from said first viewpoint position are present, said method further 
comprises the selection step of receiving a plurality of videos of the real space viewed from a 
first viewpoint position from the plurality of means for sensing a video viewed from said first 
viewpoint position, and outputting the video of the real space viewed from a first viewpoint 
position input from one selected means for sensing a video of said first viewpoint position to 
means for compositing a first viewpoint video, and said first video composition step includes the 
step of generating a video of the virtual object viewed from said first viewpoint position using 
parameters of the means for sensing a video viewed from a first viewpoint position selected in 
the selection step. 



Application/Control Number: 09/658,463 Page 18 

Art Unit: 2672 

Claim 18 recites all the limitations of claim 10 and adds the limitation of "selection step 
for receiving a plurality of videos." 

Daily teaches selecting of the image regions as captured by a plurality of video cameras 
by zooming in and out the video scene in the real space. Daily also teaches a plurality of video 
cameras for sensing a scene below or around the plane where image signals that represent an 
environment scene of a real space and the user's zooming in and out video scene {figure 1, 9-10; 
column 3-5; column 7-12), 



4. Claims 28-29: 

Referring to claims 28-29, Daily discloses a mixed reality presentation system that 
generates and presents a virtual image in a real space {figure l t 9-10; column 3-5; column 10- 
12). However, the reference is silent on a printing means in connection to the mixed reality 
presentation apparatus. It is common that a computer system has a printing means attached to 
them. Therefore, it would have been obvious to one having ordinary skill in the art to have 
incorporated a printing means in the mixed reality presentation apparatus of Daily because such 
construction is rather conventional. A person of ordinary skill in the art would be motivated to 
have incorporated a printing means to the Daily's mixed reality presentation apparatus to further 
provide a paper copy of still images of the real images that have been displayed on the display 
devices of Daily. 



5. Claim 31: 
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The claim 31 encompasses the same scope of invention as that of claim 1. The claim is 
subject to the same reasons set forth in claim 1. 

6. Claim 32: 

The claim 32 encompasses the same scope of invention as that of claim 1. The claim is 
subject to the same reasons set forth in claim 1 . 

Claim Rejections - 35 USC §103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

7. Claims 19 and 24-27 rejected under 35 U.S.C. 103(a) as being unpatentable over Daily 
U.S. Patent No. 6,317,127 (Daily) in view of Latypov U.S. Patent No. 6,624,853 (Latypov) and 
Sato U.S. Patent No. 6,445,815 (hereinafter Sato). 

8. Claim 19: 

(1) The claim 19 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of "a storage medium storing a program code. 

(2) It is unclear whether Daily has implicitly taught a storage medium storing a program 

code. 
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(3) However, Sato discloses the claimed limitation of a storage medium that stores an 
image processing program, which is implemented on a computer and continuously presents 
three-dimensional images to an observer/player, storing a program code of an augmented reality, 
a depth estimation program code, a depth image generation program code, a position/posture 
information estimation program code, a warping program code and a program code of presenting 
to the observer three-dimensional images. The Sato reference implicitly teaches a program code 
including the augmented reality presentation step of generating and presenting a virtual image in 
a real space in which an image of merged real objects and virtual objects are displayed in LCDs 
103 in an optical see-through head mounted device (column 1, lines 13-23). The Sato reference 
also implicitly teaches a position/posture estimation module 201 that outputs three-dimensional 
motions from a viewpoint position of the camera to right and left viewpoint positions of the 
player (column 12, lines 54-65). The Sato reference further teaches a depth image generation 
module 300 that uses position/posture information input from the position/posture estimation 
module 201 as that for CG rendering and which generates an augmented reality image using the 
three-dimensional CG database in accordance with the distance to an object in the real world 
expressed by the depth image and presents it on the LCDs 103 (column 12, lines 44-50) and a 
depth warping module 203 to inversely project a depth image ID acquired at a viewpoint having 
position/posture information into a space, and to re-project it onto the imaging plane of the 
virtual camera with the focal length of the virtual camera assumed at the viewpoint having an 
estimated position/posture value by the viewpoint position/posture module 201 (column 13, lines 
32-48). 
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(4) Daily at least suggests program codes illustrated in a variety of flow charts (Figs. 5- 

9). 

(5) One having the ordinary skill in the art would have been motivated to do this because 
it would have provided the installation and implementation of the image processing on other 
systems. 

Claim 24: 

The medium according to claim 19, characterized in that parameters of means for 
sensing said first viewpoint video are known, and the program code of said first video generation 
step includes the step of generating the video of the virtual object viewed from said first 
viewpoint position in accordance with the known parameters. 

Claim 24 recites all the limitations of claim 19 and adds the limitation of "the known 
parameters." Daily teaches the known parameters such as the focal length of the CCD video 
camera and measured plane's coordinates by GPS sensors, and the video scenes can be adjusted 
in accordance to the desired resolution, the object's focal length, orientation and Cartesian 
coordinates on the object's image plane where the object is either the image source or the user 
display {figure 1, 9-10; column 3-5; column 7-12). 

Claim 25: 

The medium according to claim 19, characterized in that some of parameters of means 
for sensing a video viewed from said first viewpoint position are variable, the program code of 
said medium further comprises the measurement step of measuring changes of the parameters, 
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and the program code of said first video generation step includes the step of generating the video 
of the virtual object viewed from said first viewpoint position in accordance with the parameters 
measured in the measurement step. 

Claim 25 recites all the limitations of claim 19 and adds the limitation of "variable 
parameters" and "measurement step." Daily teaches measuring the plane's coordinates which are 
variables as plane moves. Daily further teaches correcting view position and posture of the 
passengers and zooming in and out the video scene {figure 1 } 9-10; column 3-5; column 7-12). 

Claim 26: 

The medium according to claim 25, characterized in that the parameters of the means of 
sensing a video viewed from said first viewpoint position measured in the measurement step 
include at least one of a viewpoint position/posture, and zoon ratio. 

Claim 26 recites all the limitations of claim 25 and adds the limitation of 
"position/posture information." Daily teaches the plane's positions are variables of the plane's 
coordinates which determine the position/posture of the CCD video cameras and the zoom ratios 
are determined by the zooming in and out parameters inputted by the user. Daily also teaches 
generating and correcting view position and posture of the passengers {figure 1, 9-10; column 3- 
5; column 7-12). 

Claim 27: 

The medium according to claim 19, characterized in that when a plurality of means for 
sensing a video viewed from said first viewpoint position are present, said medium further 
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comprises a program code of the selection step of receiving a plurality of videos of the real 
space viewed from a first viewpoint position from the plurality of means for sensing a video 
viewed from said first viewpoint position, and outputting the video of the real space viewed from 
a first viewpoint position input from one selected means for sensing a video of said first 
viewpoint position to means for compositing a first viewpoint video, and the program code of 
said first video composition step includes the step of generating a video of the virtual object 
viewed from said first viewpoint position using parameters of the means for sensing a video 
viewed from a first viewpoint position selected in the selection step. 

Claim 27 recites all the limitations of claim 19 and adds the limitation of "selection step 
for receiving a plurality of videos." 

Daily teaches selecting of the image regions as captured by a plurality of video cameras 
by zooming in and out the video scene in the real space. Daily also teaches a plurality of video 
cameras for sensing a scene below or around the plane where image signals that represent an 
environment scene of a real space and the user's zooming in and out video scene (figure 1, 9-10; 
column 3-5; column 7-12). 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
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the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (571) 272-7665. 
The examiner can normally be reached on 8:00 - 6:30 (Mon-Thu). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (571) 272-7664. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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